Add a test for CUDA library build rules #40605

fwyzard · 2023-01-24T17:33:49Z

PR description:

This PR adds new packages to test the implementation and usage of CUDA libraries for device code in CMSSW plugins and executables.

The package HeterogeneousTest/CUDADevice implements a library that defines and exports CUDA device-only functions, and a plugin and test that use them.

The package HeterogeneousTest/CUDAKernel implements a library that imports device functions from HeterogeneousTest/CUDADevice to define and export CUDA kernels, and a plugin and test that use them.

The package HeterogeneousTest/CUDAWrapper implements a library that imports kernels from HeterogeneousTest/CUDAKernel to define and export host-only wrappers around them, usable by non-CUDA libraries, plugins and applications, and implements a plugin and test that use them.

The package HeterogeneousTest/CUDAOpaque implements a library that use the wrappers from HeterogeneousTest/CUDAKernel to define and export host-only functions around the whole CUDA section, usable by libraries, plugins and applications that are not CUDA-aware, and implements a plugin and test that use them.

PR validation:

The new unit tests compile and pass.

fwyzard · 2023-01-24T17:34:35Z

@makortel this almost works - it needs only a fix in the build command.

fwyzard · 2023-01-24T17:38:24Z

@smuzaffar at least locally, this fails to link because scram does not pass -lHeterogeneousCoreCUDATestDeviceLib to gcc, even if the build file does state

    <use name="HeterogeneousCore/CUDATestDeviceLib"/>

The resulting command is

>> Building binary testCudaDeviceAddition
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02768/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/bin/c++ -O2 -pthread -pipe -Werror=main -Werror=pointer-arith -Werror=overlength-strings -Wno-vla -Werror=overflow -std=c++1z -ftree-vectorize -Werror=array-bounds -Werror=format-contains-nul -Werror=type-limits -fvisibility-inlines-hidden -fno-math-errno --param vect-max-version-for-alias-checks=50 -Xassembler --compress-debug-sections -msse3 -felide-constructors -fmessage-length=0 -Wall -Wno-non-template-friend -Wno-long-long -Wreturn-type -Wextra -Wpessimizing-move -Wclass-memaccess -Wno-cast-function-type -Wno-unused-but-set-parameter -Wno-ignored-qualifiers -Wno-deprecated-copy -Wno-unused-parameter -Wunused -Wparentheses -Wno-deprecated -Werror=return-type -Werror=missing-braces -Werror=unused-value -Werror=unused-label -Werror=address -Werror=format -Werror=sign-compare -Werror=write-strings -Werror=delete-non-virtual-dtor -Werror=strict-aliasing -Werror=narrowing -Werror=unused-but-set-variable -Werror=reorder -Werror=unused-variable -Werror=conversion-null -Werror=return-local-addr -Wnon-virtual-dtor -Werror=switch -fdiagnostics-show-option -Wno-unused-local-typedefs -Wno-attributes -Wno-psabi -DUSE_CMS_DEPRECATED -DBOOST_DISABLE_ASSERTS -fPIC tmp/el8_amd64_gcc11/src/HeterogeneousCore/CUDATestDeviceLib/test/testCudaDeviceAddition/testDeviceAddition.cu.o tmp/el8_amd64_gcc11/src/HeterogeneousCore/CUDATestDeviceLib/test/testCudaDeviceAddition/testCudaDeviceAddition_cudadlink.o -Wl,-E -Wl,--hash-style=gnu -L/data/user/fwyzard/CMSSW_13_0_X_2023-01-21-1100/biglib/el8_amd64_gcc11 -L/data/user/fwyzard/CMSSW_13_0_X_2023-01-21-1100/lib/el8_amd64_gcc11 -L/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-01-21-1100/biglib/el8_amd64_gcc11 -L/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-01-21-1100/lib/el8_amd64_gcc11 -L/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-01-21-1100/external/el8_amd64_gcc11/lib -L/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02768/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_X_2023-01-19-2300/lib/el8_amd64_gcc11 -L/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02768/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_X_2023-01-19-2300/biglib/el8_amd64_gcc11 -L/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02768/el8_amd64_gcc11/external/cuda/11.5.2-66a9473808e7d5863d5bbec0824e2c4a/lib64/stubs -lcudart -lcudadevrt -lnvToolsExt -lcuda -o tmp/el8_amd64_gcc11/src/HeterogeneousCore/CUDATestDeviceLib/test/testCudaDeviceAddition/testCudaDeviceAddition

and it fails with

/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02768/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/bin/../lib/gcc/x86_64-redhat-linux-gnu/11.2.1/../../../../x86_64-redhat-linux-gnu/bin/ld: tmp/el8_amd64_gcc11/src/HeterogeneousCore/CUDATestDeviceLib/test/testCudaDeviceAddition/testCudaDeviceAddition_cudadlink.o: in function `__cudaRegisterLinkedBinary_c84dd51f_17_DeviceAddition_cu_bcd44480':
link.stub:(.text+0x121): undefined reference to `__fatbinwrap_c84dd51f_17_DeviceAddition_cu_bcd44480'
collect2: error: ld returned 1 exit status

If I add by by hand -lHeterogeneousCoreCUDATestDeviceLib the link is successful, and the test actually runs:

$ ./tmp/el8_amd64_gcc11/src/HeterogeneousCore/CUDATestDeviceLib/test/testCudaDeviceAddition/testCudaDeviceAddition
===============================================================================
All tests passed (2097152 assertions in 1 test case)

fwyzard · 2023-01-24T17:38:50Z

please test

cmsbuild · 2023-01-24T17:41:38Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40605/33862

This PR adds an extra 12KB to repository

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40605/33862/code-format.patch
e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40605/33862/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

smuzaffar · 2023-01-24T22:27:46Z

@fwyzard , cms-sw/cmsdist#8264 should fix this issue. Problem was that there was a restrictions that only packages with at least one cxx files can export libs. In this case HeterogeneousCore/CUDATestDeviceLib has only .cu files that is why its lib was not exported. cms-sw/cmsdist#8264 now allows packages with any source file (cxx, c, fortran, cuda) should export their libs

fwyzard · 2023-01-25T04:29:39Z

Understood... thanks!

fwyzard · 2023-01-25T05:08:53Z

please test with cms-sw/cmsdist#8264

cmsbuild · 2023-01-25T05:12:50Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40605/33864

This PR adds an extra 12KB to repository

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40605/33864/code-format.patch
e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40605/33864/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

cmsbuild · 2023-01-31T21:03:27Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40605/33979

This PR adds an extra 16KB to repository

cmsbuild · 2023-01-31T21:03:58Z

Pull request #40605 was updated. @makortel, @fwyzard can you please check and sign again.

fwyzard · 2023-01-31T21:40:02Z

+heterogeneous

cmsbuild · 2023-01-31T21:40:30Z

This pull request is fully signed and it will be integrated in one of the next master IBs after it passes the integration tests. This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

cmsbuild · 2023-01-31T23:29:41Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8dc05e/30294/summary.html
COMMIT: 816674f
CMSSW: CMSSW_13_0_X_2023-01-31-1100/el8_amd64_gcc11
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/40605/30294/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 5 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3555495
DQMHistoTests: Total failures: 3
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3555470
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 211 log files, 162 edm output root files, 49 DQM output files
TriggerResults: no differences found

GPU Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 9 differences found in the comparisons
DQMHistoTests: Total files compared: 4
DQMHistoTests: Total histograms compared: 19862
DQMHistoTests: Total failures: 272
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 19590
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 3 files compared)
Checked 12 log files, 9 edm output root files, 4 DQM output files
TriggerResults: found differences in 2 / 3 workflows

rappoccio · 2023-02-01T07:48:39Z

+1

Tests are now successful.

cmsbuild added this to the CMSSW_13_0_X milestone Jan 24, 2023

cmsbuild added code-checks-pending new-package-pending orp-pending pending-signatures tests-pending labels Jan 24, 2023

cmsbuild added tests-started and removed tests-pending labels Jan 24, 2023

cmsbuild added code-checks-rejected and removed code-checks-pending labels Jan 24, 2023

smuzaffar mentioned this pull request Jan 24, 2023

[BuildRules] Allow packages to export lib even if there are no cxx files cms-sw/cmsdist#8264

Merged

fwyzard force-pushed the CUDATestDeviceLib branch from 43491cb to ba444f4 Compare January 25, 2023 05:08

cmsbuild added code-checks-pending tests-pending and removed tests-started code-checks-rejected labels Jan 25, 2023

cmsbuild added requires-external tests-started and removed tests-pending labels Jan 25, 2023

cmsbuild added code-checks-rejected and removed code-checks-pending labels Jan 25, 2023

fwyzard force-pushed the CUDATestDeviceLib branch from ba444f4 to e30732b Compare January 25, 2023 05:14

cmsbuild added pending-signatures tests-started code-checks-pending heterogeneous-pending and removed fully-signed tests-approved code-checks-approved heterogeneous-approved labels Jan 31, 2023

cmsbuild added code-checks-approved and removed code-checks-pending labels Jan 31, 2023

cmsbuild added fully-signed heterogeneous-approved and removed pending-signatures heterogeneous-pending labels Jan 31, 2023

cmsbuild added tests-approved and removed tests-started labels Jan 31, 2023

smuzaffar mentioned this pull request Feb 1, 2023

[BuildRules] set TMPDIR so that hipcc can create temp file in cmssw/tmp/package directory cms-sw/cmsdist#8279

Merged

cmsbuild added orp-approved and removed orp-pending labels Feb 1, 2023

cmsbuild merged commit e86ddad into cms-sw:master Feb 1, 2023

This was referenced Feb 1, 2023

Update Tensorflow to version 2.11.0 cms-sw/cmsdist#8258

Closed

[Tensorflow] Build with GPU enabled cms-sw/cmsdist#7648

Merged

Custom Run 3 PFScouting NanoAOD #40438

Merged

fwyzard deleted the CUDATestDeviceLib branch February 2, 2023 12:44

Add a test for CUDA library build rules #40605

Add a test for CUDA library build rules #40605

Uh oh!

Conversation

fwyzard commented Jan 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR description:

PR validation:

Uh oh!

fwyzard commented Jan 24, 2023

Uh oh!

fwyzard commented Jan 24, 2023

Uh oh!

fwyzard commented Jan 24, 2023

Uh oh!

cmsbuild commented Jan 24, 2023

Uh oh!

smuzaffar commented Jan 24, 2023

Uh oh!

fwyzard commented Jan 25, 2023

Uh oh!

fwyzard commented Jan 25, 2023

Uh oh!

cmsbuild commented Jan 25, 2023

Uh oh!

cmsbuild commented Jan 31, 2023

Uh oh!

cmsbuild commented Jan 31, 2023

Uh oh!

fwyzard commented Jan 31, 2023

Uh oh!

cmsbuild commented Jan 31, 2023

Uh oh!

cmsbuild commented Jan 31, 2023

Comparison Summary

GPU Comparison Summary

Uh oh!

rappoccio commented Feb 1, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

fwyzard commented Jan 24, 2023 •

edited

Loading